Remove opendal dependency from QEMU driver#535
Remove opendal dependency from QEMU driver#535ambient-code[bot] wants to merge 5 commits intomainfrom
Conversation
❌ Deploy Preview for jumpstarter-docs failed. Why did it fail? →
|
|
Important Review skippedBot user detected. To trigger a single review, invoke the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Fixed the CI failures by adding The new The version requirement |
| @@ -0,0 +1,286 @@ | |||
| """ | |||
| Simplified FlasherClient that does not depend on opendal. | |||
There was a problem hiding this comment.
I think it's reasonable to start accumulating common driver interfaces in the core jumpstarter, as, for example, this one is broadly used around Jumpstarter, and lived on opendal initially.
There was a problem hiding this comment.
Agreed! The FlasherInterface ABC is purely abstract with no external dependencies, so it makes sense for it to live in core. This way any driver can implement it without pulling in opendal as a transitive dependency.
| local_path, url = _parse_path(path) | ||
|
|
||
| if url is not None: | ||
| with _http_url_adapter(client=self, url=url, mode="wb") as handle: |
There was a problem hiding this comment.
Are we testing all paths in here: https://github.com/jumpstarter-dev/jumpstarter/blob/main/python/packages/jumpstarter-driver-qemu/jumpstarter_driver_qemu/driver_test.py#L79 ?
I see it uses the Operator, can we make sure that we take that and interpret that for compatibility purposes without really relying/using opendal?
There was a problem hiding this comment.
Good question. The current test covers two paths:
- Local file path (
cached_image.resolve()) -- this exercises the_local_file_adaptercode path in the newFlasherClient, streaming the file viaresource_async. - HTTP URL (the Fedora download URL) -- this was previously using
Operator("http", endpoint=...)and is now a directhttps://URL, which exercises the_http_url_adaptercode path usingPresignedRequestResource.
Both paths in the new FlasherClient are covered by the integration test.
Regarding backward compatibility with opendal.Operator: the opendal package's FlasherClient (in jumpstarter_driver_opendal.client) is unchanged and still accepts the operator= parameter. Drivers that depend on jumpstarter-driver-opendal (esp32, pi-pico, dutlink, sdwire) continue to use that client via their own FlasherInterface.client() classmethod which points to the opendal version.
The QEMU driver now uses the core FlasherInterface which points to the simplified FlasherClient -- this one intentionally does not accept an operator= parameter since the goal is to avoid the opendal dependency entirely. The Operator usage in the old test was only needed because the old flasher client required it for HTTP downloads; the new client handles HTTP URLs natively.
So the API surface for QEMU users changes from:
# Old (required opendal)
qemu.flasher.flash("path/on/server", operator=Operator("http", endpoint="https://example.com"))
# New (no opendal needed)
qemu.flasher.flash("https://example.com/path/on/server")The new API is simpler and doesn't break any existing QEMU usage since the operator= parameter was only used in this test, not in production QEMU workflows.
There was a problem hiding this comment.
Ok, but if that test is there, is not executing. Should we make sure to have a test that executes the full url download?
There was a problem hiding this comment.
You're right -- when the cached image exists (which it does in CI), the HTTP URL branch is never exercised.
I'll add a dedicated unit test for the _http_url_adapter and _parse_path helpers in the core jumpstarter package to make sure the URL code path is covered without needing a full Fedora image download. This will verify that:
_parse_pathcorrectly distinguishes local paths from HTTP/HTTPS URLs_http_url_adapterproduces a properPresignedRequestResourcefor HTTP URLs- The
FlasherClient._flash_singlemethod routes HTTP URLs through the presigned request path
For a full end-to-end HTTP download test, that would involve downloading a large image in CI which is expensive. I think a unit test covering the routing logic is the right trade-off here. Does that work for you?
|
Added unit tests for The new tests in
This ensures the URL download logic is exercised in CI without needing to download a full Fedora image. |
|
Fixed the
The netlify deploy preview failure is unrelated to this PR's changes. |
|
Rebase the pr please. |
6c8eee9 to
bffbd65
Compare
|
Rebased onto main (d7a65e0). The branch now has a clean history on top of the latest main. |
|
Rebase complete. The branch |
| def flash(self, source, target: str | None = None): ... | ||
|
|
||
| @abstractmethod | ||
| def dump(self, target, partition: str | None = None): ... |
There was a problem hiding this comment.
[MEDIUM] Parallel FlasherInterface ABCs with no inheritance relationship
There are now two identical FlasherInterface ABCs -- this one in core and the one in jumpstarter-driver-opendal -- with no inheritance link between them. Both define flash() and dump() with the same signatures but different client() return values. Any future contract change must be manually synchronized.
Since this is phase 1 of the incremental migration (#441), consider having the opendal FlasherInterface inherit from this core one in a follow-up, or plan to remove the opendal copy once remaining drivers migrate.
AI-generated, human reviewed
There was a problem hiding this comment.
Agreed -- this duplication is intentional for phase 1 of #441 to keep this PR focused on removing the opendal dependency from QEMU without touching the other drivers.
For the follow-up, I think the cleanest approach is to have the opendal FlasherInterface inherit from this core one (they already have identical flash()/dump() signatures). That way any future contract change only needs to happen in one place. I'll track this in #441.
|
|
||
| http_patch.assert_called_once_with(client=client, url="https://example.com/dump.bin", mode="wb") | ||
| local_patch.assert_not_called() | ||
| call_mock.assert_called_once_with("dump", "http_handle", None) |
There was a problem hiding this comment.
[MEDIUM] No test coverage for _local_file_adapter streaming behavior
The tests cover path parsing and routing logic via mocks, but the actual _local_file_adapter streaming pipeline is never exercised directly. _AsyncIteratorStream and _FileWriteObjectStream contain non-trivial logic (lazy file open, chunked reading, progress attributes) that could break silently. Consider adding at least a focused unit test that exercises _local_file_adapter with a real temp file to verify the streaming and chunking behavior end-to-end.
AI-generated, human reviewed
There was a problem hiding this comment.
Good point. The streaming internals (_AsyncIteratorStream, _FileWriteObjectStream, chunked reading) are non-trivial and currently only tested indirectly through the routing mocks.
I'd prefer to add a focused streaming integration test as a follow-up rather than in this PR, since it would need to set up a mock DriverClient with a working resource_async pipeline, which adds meaningful complexity. The routing tests give us confidence the right code paths are selected, and the e2e QEMU tests exercise the full pipeline end-to-end. Will track this as a follow-up to #441.
| For local files: streams via the existing resource_async mechanism using anyio. | ||
| For HTTP URLs: passes a PresignedRequestResource directly to the exporter, | ||
| which already handles presigned downloads via aiohttp. | ||
| """ |
There was a problem hiding this comment.
[MEDIUM] Module-level docstrings may violate project comment policy
This file, driver/flasher.py, and flasher_test.py all contain module-level docstrings explaining the implementation approach. The project rules state comments should be a last resort when code cannot be self-explanatory. Existing comparable modules (e.g., opendal/client.py, opendal/driver.py) do not use module-level docstrings. The module names, class names, and method signatures already convey purpose -- consider removing these docstrings to stay consistent with the rest of the codebase.
AI-generated, human reviewed
There was a problem hiding this comment.
Fair point -- removed all three module-level docstrings in 0c2d55c to match the rest of the codebase. The module/class/method names are self-explanatory.
| pass | ||
|
|
||
| async def aclose(self): | ||
| pass |
There was a problem hiding this comment.
[LOW] _AsyncIteratorStream.aclose() does not propagate close to wrapped async generator
The aclose() method is a no-op, so if the stream is abandoned before the async generator completes (e.g., on error), the file handle inside the generator may not be closed promptly. Adding await self.iterator.aclose() would ensure the async with block in the generator properly closes the file handle. Impact is minor (only affects error paths, GC will eventually clean up), but it would be cleaner.
async def aclose(self):
await self.iterator.aclose()AI-generated, human reviewed
There was a problem hiding this comment.
Good catch. Fixed in 0c2d55c -- aclose() now calls await self.iterator.aclose() which will properly trigger the async with cleanup in the generator and close the file handle on error paths.
|
|
||
| http_patch.assert_called_once_with(client=client, url="https://example.com/dump.bin", mode="wb") | ||
| local_patch.assert_not_called() | ||
| call_mock.assert_called_once_with("dump", "http_handle", None) |
There was a problem hiding this comment.
[LOW] No test for dict-based multi-target flash
FlasherClient.flash() supports passing dict[str, PathBuf] for multi-target flashing (around lines 258-267 of flasher.py). No test covers this code path, including the ArgumentError case when both a dict and a target kwarg are provided. Adding a test for the happy path and the error case would improve confidence in this feature.
AI-generated, human reviewed
There was a problem hiding this comment.
Agreed this would be good coverage. The dict-based multi-target flash path (including the ArgumentError guard) is straightforward but untested. I'll add tests for both the happy path and the error case as a follow-up -- keeping this PR focused on the core changes to avoid scope creep.
|
|
||
| http_patch.assert_called_once_with(client=client, url="https://example.com/dump.bin", mode="wb") | ||
| local_patch.assert_not_called() | ||
| call_mock.assert_called_once_with("dump", "http_handle", None) |
There was a problem hiding this comment.
[LOW] No test for dump with local path routing
TestFlasherClientRouting tests dump routing for HTTP URLs but has no corresponding test for dump routing to local file paths. Adding a test_dump_local_path case would round out the routing coverage.
AI-generated, human reviewed
There was a problem hiding this comment.
Added test_dump_routes_local_path in 0c2d55c -- this mirrors the existing test_dump_routes_http_url test and verifies that local paths go through _local_file_adapter with the correct mode="wb" parameter.
|
Addressed @raballew's review feedback in Fixed in this commit:
Tracked for follow-up in #441:
|
Move FlasherInterface to core jumpstarter package (jumpstarter.driver.flasher) and create a simplified FlasherClient (jumpstarter.client.flasher) that streams local files via resource_async and passes HTTP URLs as PresignedRequestResource for exporter-side download, eliminating the need for the opendal library. Update QEMU driver to import from the new core location and remove jumpstarter-driver-opendal from its dependencies. Update tests to use direct HTTP URLs instead of opendal Operator. Closes #441 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The new FlasherClient in jumpstarter/client/flasher.py imports click but it was not listed in the dependencies, causing import errors in tests. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Ensures the HTTP URL code path in FlasherClient is covered by tests, since the integration test only exercises this path when no cached image is available. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove unused `pathlib.Path` and `PresignedRequestResource` imports that were flagged by ruff (F401). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…l test - Remove module-level docstrings from flasher.py, driver/flasher.py, and flasher_test.py to match project conventions (per raballew review) - Fix _AsyncIteratorStream.aclose() to propagate close to wrapped async generator, ensuring file handles are cleaned up on error paths - Add test_dump_routes_local_path to cover dump routing for local file paths Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
0c2d55c to
69cbb0e
Compare
| class FlasherInterface(metaclass=ABCMeta): | ||
| @classmethod | ||
| def client(cls) -> str: | ||
| return "jumpstarter.client.flasher.FlasherClient" | ||
|
|
||
| @abstractmethod | ||
| def flash(self, source, target: str | None = None): ... | ||
|
|
||
| @abstractmethod | ||
| def dump(self, target, partition: str | None = None): ... |
There was a problem hiding this comment.
[MEDIUM] Still not addressed from previous review: The new core FlasherInterface and the existing jumpstarter_driver_opendal.driver.FlasherInterface are parallel ABCs with identical method signatures but no inheritance relationship. This means drivers switching from opendal's FlasherInterface to the core one (like QEMU did) are making a silent, potentially breaking change for any code doing isinstance checks. The opendal FlasherInterface should be updated to inherit from the core one, or at minimum re-export it, to establish a clear hierarchy.
AI-generated, human reviewed
| @dataclass(kw_only=True) | ||
| class _AsyncIteratorStream(ObjectStream[bytes]): | ||
| """Wraps an async iterator as an ObjectStream for resource_async.""" | ||
|
|
||
| iterator: Any | ||
| total: int | None = None | ||
|
|
||
| async def receive(self) -> bytes: | ||
| try: | ||
| return await self.iterator.__anext__() | ||
| except StopAsyncIteration: | ||
| raise EndOfStream from None | ||
|
|
||
| async def send(self, item: bytes): | ||
| raise BrokenResourceError("read-only stream") | ||
|
|
||
| async def send_eof(self): | ||
| pass | ||
|
|
||
| async def aclose(self): | ||
| await self.iterator.aclose() | ||
|
|
||
| @property | ||
| def extra_attributes(self) -> Mapping[Any, Callable[[], Any]]: | ||
| if self.total is not None and self.total > 0: | ||
| return {ProgressAttribute.total: lambda: float(self.total)} | ||
| return {} |
There was a problem hiding this comment.
[MEDIUM] Still not addressed from previous review: _AsyncIteratorStream and _FileWriteObjectStream have no direct unit tests exercising their streaming behavior (receive/send/aclose lifecycle, error propagation on partial reads, EndOfStream on empty files, etc.). The existing routing tests mock these classes entirely, so actual I/O paths remain uncovered.
AI-generated, human reviewed
|
|
||
| local_patch.assert_called_once() | ||
| http_patch.assert_not_called() | ||
| call_mock.assert_called_once_with("dump", "local_handle", None) |
There was a problem hiding this comment.
[LOW] Still not addressed from previous review: No test for dict-based multi-target flash. FlasherClient.flash accepts a dict[str, PathBuf] and iterates over entries calling _flash_single for each, and also raises ArgumentError when both a dict and target are provided. Neither path is tested.
AI-generated, human reviewed
| if url is not None: | ||
| # HTTP URL: pass as presigned request for exporter-side download | ||
| with _http_url_adapter(client=self, url=url, mode="rb") as handle: | ||
| return self.call("flash", handle, target) |
There was a problem hiding this comment.
[MEDIUM] The compression parameter is silently ignored when the path is an HTTP URL. _http_url_adapter does not accept a compression keyword, so _flash_single drops it without warning when routing to the HTTP path. The same applies to dump at lines 273-275. Either raise an error/warning when compression is specified with an HTTP URL, or pass it through to the exporter.
AI-generated, human reviewed
| """Dump image from DUT""" | ||
| local_path, url = _parse_path(path) | ||
|
|
||
| if url is not None: | ||
| with _http_url_adapter(client=self, url=url, mode="wb") as handle: | ||
| return self.call("dump", handle, target) | ||
| else: | ||
| with _local_file_adapter(client=self, path=local_path, mode="wb", compression=compression) as handle: | ||
| return self.call("dump", handle, target) |
There was a problem hiding this comment.
[LOW] Parameter naming is inconsistent between the driver-side FlasherInterface and the client-side FlasherClient. The driver's dump(self, target, partition) uses target for the destination resource and partition for the optional selector, while this client's dump(self, path, *, target) uses path for the destination and target for the selector. Since the client maps its target kwarg to the driver's partition positionally via self.call("dump", handle, target), this works at runtime but creates a confusing API surface. Consider aligning the naming.
AI-generated, human reviewed
Summary
FlasherInterfaceABC to corejumpstarterpackage (jumpstarter.driver.flasher) — it's a pure abstract class with no opendal importsFlasherClientin core (jumpstarter.client.flasher) that handles local files viaresource_asyncand HTTP URLs viaPresignedRequestResource, without requiring the opendal libraryFlasherInterfacefromjumpstarter.driverinstead ofjumpstarter_driver_opendal.driverjumpstarter-driver-opendalfrom QEMU'spyproject.tomldependenciesopendal.OperatorBackward Compatibility
FlasherInterfaceis unchanged — drivers that depend on opendal (esp32, pi-pico, dutlink, sdwire) continue to work without modificationFlasherInterface.client()points tojumpstarter.client.flasher.FlasherClient(simplified, no opendal)FlasherInterface.client()continues pointing tojumpstarter_driver_opendal.client.FlasherClient(full opendal support)Test plan
test_driver_qemu) fails only due to missingqemu-imgbinary in CI environment (pre-existing)Related #441
🤖 Generated with Claude Code